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Background/context: 

Description of prior research, its intellectual context and its policy context. 



Treatment fidelity (i.e., integrity) can be defined as the extent to which an intervention is 
delivered as intended. Inevitable slippage between the specified and implemented intervention 
means that any treatment effects must be attributed not to the theoretical intervention model but 
to the model as implemented. Ideally, fidelity assessments would follow a specific course, 
beginning with a full characterization of the intervention “in theory” by outlining: the constructs 
and processes being manipulated (change model), the essential components of the intervention 
and required support services (logic model), and the context of implementation within the school 
year (operational model). Explication of these models leads to the development of reliable and 
valid measures of achieved intervention fidelity (Cordray & Pion, 2006), and the integration of 
fidelity measures into outcomes analysis. Without measuring achieved fidelity in both treatment 
and control conditions we are unable to determine whether differences in outcomes (or lack 
thereof) are due to achieved fidelity in the treatment conditions or the presence of core 
intervention components in the control condition. An intervention may have a high degree of 
achieved fidelity, but fail to have effects because it does not differ from the control conditions on 
core intervention components (i.e., small achieved relative strength). 

Purpose / objective / research question / focus of study: 

Description of what the research focused on and why. 



This paper outlines a method of determining the achieved relative strength of an 
intervention by calculating the extent to which core intervention components are present in both 
the treatment and control condition (Hulleman & Cordray, 2009). Using data from several 
examples we demonstrate how to calculate ARS indices, and subsequently how to incorporate 
them into analyses of intervention effectiveness. 

Setting: 

Description of where the research took place. 



The results from three different samples are reported in this paper. Sample 1 was 
collected in an experimental laboratory on a college campus. Sample 2 was collected within an 
undergraduate class. Sample 3 was obtained from secondary data analysis of Reading First data. 

Population / Participants / Subjects: 

Description of participants in the study: who (or what ) how many, key features (or characteristics). 



Sample 1: This sample consists of introductory psychology students at a major mid- 
Western university who participated in the experiment for course credit. Participants volunteered 
to visit our research laboratory and participate in a randomized experiment. 
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Sample 2: This sample consists of introductory psychology students at a major mid- 
Western university who participated in the experiment for course credit. Students completed the 
experimental intervention as part of an introductory psychology course assignment. 

Sample 3: This sample consists of secondary data analysis from the Reading First 
Implementation report (U.S. Department of Education, 2008). 

Intervention / Program / Practice: 

Description of the intervention, program or practice, including details of administration and duration. 



The randomized intervention in the first two samples (the psychology laboratory and the 
college classroom) was a curricular intervention designed to encourage participants to make 
connections between the material they were studying and their own lives (for a more complete 
description of the intervention see Hulleman & Cordray, 2009; Hulleman, Godes, Hendricks, & 
Harackiewicz, in press; and Hulleman & Harckiewicz, in press). In both studies students were 
randomly assigned to the treatment group or the control group. Treatment participants selected a 
topic they were studying (either mental math in the laboratory or a topic from their introductory 
psychology class in college classroom) and wrote about how it connected to their lives, or to the 
lives of someone they knew. In the control condition, participants selected a topic and wrote a 
summary of what they had been learning in the laboratory or the classroom. 

Reading First is a policy intervention intended to ensure all children can read at grade 
level or above. Reading First provides funds to Title I schools to increase the amount of teacher 
professional development to K-3 teachers regarding scientifically-based reading instruction. The 
exact nature of the additional resources is up to each individual school to determine. For more 
details see U.S. Department of Education (2008). 

Research Design: 

Description of research design (e.g., qualitative case study, quasi-experimental design, secondary analysis, analytic 
essay, randomized field trial). 



All three samples utilized randomized control trials. Sample 1 and 2 were randomized at 
the student level, whereas the Reading First sample a nationally-representative sample of 
Reading First schools and non-Reading First Title I schools (U.S. Department of Education, 
2008). 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 



Data was collected individually from participants in Samples 1 and 2. Data were analyzed 
using multiple regression. In the Reading First sample, a nationally-representative sample of 
Reading First and non-Reading First Title I schools were collected (Moss et al., 2008). We used 
the raw data from the implementation report (e.g., means and SD’s, frequencies, etc.) to calculate 
our indices of achieved relative strength. Achieved relative strength (ARS) indices are calculated 
as the degree of fidelity within the treatment group minus the degree of fidelity within the control 
group divided by the pooled standard deviation (see Hulleman & Cordray, 2009, for more details 
on calculations). ARS indices represent the standardized difference in treatment implementation 
between the treatment and control group, and as such are indicative of relative treatment strength 
(Cordray & Pion, 2006). 
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Findings / Results: 

Description of main findings with specific details. 



In two randomized control trials of a motivational intervention, one in the laboratory and 
the other in the classroom, we demonstrate that the ARS indices are positively correlated with 
student motivational outcomes. In secondary analyses of Reading First data we show how ARS 
indices can be created and combined in more complex, multi-component interventions. One of 
the principle findings in this sample is that implementation fidelity of Reading First core 
components is already high in the control condition, thus diminishing the impact of 
randomization to the Reading First program. 

Conclusions: 

Description of conclusions and recommendations based on findings and overall study. 

Across three randomized experiments, we demonstrated how to compute the ARS and 
how it can be utilized to understand the results of experiments. The motivation intervention 
examples demonstrate consistency across three different methods of calculating achieved relative 
strength, and how treatment fidelity diffuses when moving from more controlled to less 
controlled settings. The Reading First example demonstrates the importance of assessing core 
intervention components in control conditions as the average level of implementation of Reading 
First components in control conditions was quite high, thus diminishing the achieved relative 
strength of the intervention. 
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Appendices 
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